Evaluating Content Selection in Summarization: The Pyramid Method
نویسندگان
چکیده
We present an empirically grounded method for evaluating content selection in summarization. It incorporates the idea that no single best model summary for a collection of documents exists. Our method quantifies the relative importance of facts to be conveyed. We argue that it is reliable, predictive and diagnostic, thus improves considerably over the shortcomings of the human evaluation method currently used in the Document Understanding Conference.
منابع مشابه
Automatically Evaluating Content Selection in Summarization without Human Models
We present a fully automatic method for content selection evaluation in summarization that does not require the creation of human model summaries. Our work capitalizes on the assumption that the distribution of words in the input and an informative summary of that input should be similar to each other. Results on a large scale evaluation from the Text Analysis Conference show that input-summary...
متن کاملTJU GSummary at TAC2011: Category oriented extractive content selection for guided summarization
The guided summarization aims to capture evolving information of a single topic changing over time under the background of much emergency happenings. It delivers salient and novel information to a user who has already read a set of older documents covering the same topic. Topic is category oriented, however, there is no query description. Therefore, guided summarization raises new challenges. I...
متن کاملLearning Summary Content Units with Topic Modeling
In the field of multi-document summarization, the Pyramid method has become an important approach for evaluating machine-generated summaries. The method is based on the manual annotation of text spans with the same meaning in a set of human model summaries. In this paper, we present an unsupervised, probabilistic topic modeling approach for automatically identifying such semantically similar te...
متن کاملPower Mean Pyramid Scores for Summarization Evaluation
We present Power Mean Pyramid Scores (PMP), an evaluation metric that extends the Pyramid evaluation scheme for summarization by combining Sentence Content Units (SCU) scores using Power Mean. The Pyramid method generates a summarization score by linearly combining component SCU scores. We find that by combining SCU scores using Power Mean, we can optimize a single parameter, α, leading to sign...
متن کاملEntity-driven Rewrite for Multi-document Summarization
In this paper we explore the benefits from and shortcomings of entity-driven noun phrase rewriting for multidocument summarization of news. The approach leads to 20% to 50% different content in the summary in comparison to an extractive summary produced using the same underlying approach, showing the promise the technique has to offer. In addition, summaries produced using entity-driven rewrite...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2004